Medication identification and verification

ABSTRACT

A pill identification system includes a device which is configured to collect three-dimensional image data of surfaces of one or more pills, generate geometric features of the pill(s) from the three-dimensional image data, and identify the pill(s) using the geometric features. In some embodiments, the identified pills can be verified for patient administration by comparing the identified pills to prescription information.

RELATED APPLICATIONS

This Application is a national stage filing under 35 U.S.C. §371 ofInternational Application Serial No. PCT/IB2011/002813, entitled“MEDICATION IDENTIFICATION AND VERIFICATION” filed Oct. 27, 2011, whichwas published under PCT Article 21(2) in English, which claims priorityunder 35 U.S.C. §119(e) to U.S. Provisional Application Ser. No.61/408,289, entitled “BEDSIDE MEDICATION IDENTIFICATION ANDVERIFICATION” filed on Oct. 29, 2010, each of which is hereinincorporated by reference in its entirety.

FIELD

The present invention relates generally to the methods and apparatusesfor reducing medication errors, and more specifically to theidentification of pills via feature extraction.

DESCRIPTION OF THE RELATED ART

In healthcare institutions, nurses, doctors and pharmacists dispensemedication for patients, and typically organize dosages based on thetiming of medication administration. The medications are dispensed intoa container based on prescription information issued by the medicalstaff. Dispensing is labor intensive and error prone, and can result inmissed dosages and/or incorrect medications being dispensed. Forexample, if a patient has two prescriptions—one indicating thatmedication A should be administered at 4 pm on Mondays and the otherindicating that medication B should be administered at 4 pm everyday—then the container for 4 pm on Mondays should contain bothmedications. Common errors may result in only one medication beingpresent in the container, or the wrong number of pills for medication Aand/or B, or even the inclusion of a third medication which should notbe included.

Once the medication has been placed in the container, a nurse or othermedical professional brings the container to the patient as thescheduled administration time approaches. Depending on the workflow inthe hospital, the elapsed time from dispensation to administration canbe several hours. During this time period, a doctor may decide to changethe prescription(s) or add new prescriptions, and unless extraprecautionary steps are taken to update the contents of the medicationcontainers which have already been dispensed, medication may beadministered in accordance with an outdated prescription.

Once the container reaches the patient and the nurse determines that theadministration time has been reached, the nurse verifies that thecorrect medications are being given to the patient. Typically, thisverification is based on information the nurse can obtain from thepatient's chart which lists all of the prescriptions for the patient.This verification typically involves the nurse comparing the contents ofthe container with the prescriptions. This verification procedure isoften performed under severe time pressure. FIG. 1 shows a flowchart 100of typical steps performed from the time of a medication beingprescribed through administration of the medication to the patient.

U.S. Hospitals provide medications to approximately one million patientsdaily. Some estimates are that an average of twenty people die per daydue to medication errors in U.S. hospitals. According to the Instituteof Medicine, approximately 1.5 million people are injured by medicationerrors in the U.S. each year and the direct cost of these types ofmistakes is estimated to be $15-20 billion per year in Europe and theU.S. Ensuring that each patient receives the right dosage at the righttime can be a complicated and error-prone process. During administrationof medication in hospitals, nurses often deliver up to 100 dosageswithin thirty minutes at multiple times during the day. With a largenumber of different medications, often with similar features, on averageone medication error is made per patient per day.

Under the leadership of the Institute of Medicine and other authorities,every hospital in the western world is now aggressively seeking asolution to this missing link in medication safety. Research shows thathospitals can prevent 50% of these errors at bedside. To achieve thisreduction, nurses should be provided with tools to safely delivermedication within their tight time constraints.

One currently-used process includes placing a barcode on each individualpill prior to the pill reaching the patient administration stage. Atbedside, the nurse scans the barcode on every pill for every patientevery time medication is administered. This process can be costly,requiring an initial capital outlay of $1-2 million per hospital forvarious barcode machines and automated equipment. At present,approximately 1% of hospitals have adopted such a system.

SUMMARY

According to embodiments of the invention disclosed herein, one or moremedications are verified, often at bedside, through pill featureextraction and/or analysis. The pill features may be identified usingthree-dimensional data, such as a 3D point cloud, acquired by varioussystems and/or methods.

According to one embodiment, a system includes an imaging deviceconfigured to collect surface image data of one or more pills, and acontroller configured to control the imaging device to collect the imagedata of the one or more pills. The system is configured to generate athree-dimensional point cloud using the surface image data of the one ormore pills, and is also configured to generate geometric data for eachpill from the three-dimensional point cloud. The system is furtherconfigured to determine the identity of each of the one or more pillsbased on at least the geometric data generated from thethree-dimensional point cloud.

According to another embodiment, a method includes collecting surfaceimage data of one or more pills, generating a three-dimensional pointcloud of surface data for each of the one or more pills, and generatinggeometric data from the three-dimensional point cloud data. The methodalso includes identifying the one or more pills based on at least thegeometric data generated from the point cloud data.

According to a further embodiment, at least one computer-readablestorage medium has computer-readable instructions for performing stepsof a method of identifying a pill based at least on geometric datareceived from an imaging device. The method includes receivingthree-dimensional geometric data regarding a pill, the three-dimensionalgeometric data having been generated from a three-dimensional pointcloud of the pill. The method also includes determining that thegeometric data for the pill matches a pill geometry of a known pill, andproducing identification information regarding the pill based on atleast the determination that the that the geometric data for the pillmatches the pill geometry of a known pill.

According to yet another embodiment, a method of performing surfacescanning includes projecting a laser light pattern onto athree-dimensional surface, the laser light comprising laser light withinthe red and/or infrared spectrum, and receiving, with a camera, laserlight reflected from a portion of the three-dimensional surface. Themethod further includes using light only from a green and/or bluespectrum of the reflected light to determine the location of the portionof the three-dimensional surface from which the reflected lightreflected.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is a flowchart of a typical, known method of prescribingmedication and administering the medication to a patient;

FIG. 2 a is a front view of a medication identification apparatusaccording to one embodiment;

FIG. 2 b is a cross-sectional side view taken along line A-A in FIG. 2a;

FIG. 3 is a block diagram showing a medication identification apparatusand associated computer hardware components;

FIGS. 4 a and 4 b show a flowchart of a method of identifyingmedications, according to one embodiment;

FIGS. 5 a-5 f show several modes of a user interface for use with amedication identification apparatus, according to one embodiment;

FIG. 6 shows a flowchart of a method of collecting three-dimensionalsurface data of a pill and extracting geometric features from the data,according to one embodiment;

FIGS. 7 a-7 c are schematics of a laser scanning device configured tocapture three-dimensional data from medications;

FIG. 8 shows a flowchart of a method of collecting three-dimensionaldata, according to one embodiment;

FIG. 9 shows a flowchart of a method of capturing multi-spectral imagesof medications, according to one embodiment;

FIG. 10 shows a flowchart of a method of using a medicationidentification algorithm; and

FIG. 11 shows a flowchart of a method of training an identificationalgorithm, according to one embodiment.

DETAILED DESCRIPTION

This disclosure recognizes the importance in providing a pillidentification system which is timely, accurate and flexible. In someembodiments, a pill identification system includes a device which isconfigured to collect three-dimensional image data of surfaces of one ormore pills, generate geometric features of the pill(s) from thethree-dimensional image data, and identify the pill(s) using thegeometric features. To collect three-dimensional surface image data witha precision that permits sophisticated analysis of geometric features,various imaging techniques may be used, such as structured lightscanning and stereoscopic imaging as but two examples. These techniquespermit, in some embodiments, analysis of raised or recessed pillinscriptions, surface texture, dividing line features (such as scoreline thickness), pill volume, pill shape, and edge shapes.

For example, in some embodiments a laser scanner may be used to create a3D point cloud of pill surface data. The 3D point cloud may have asufficient density to permit the extraction of surface features from ascan of a single pill at an precision that is not possible with previoustechniques, thereby allowing accurate pill identifications.

The flexibility of a pill identification system can be important becausesome patients are administered a single pill from among over a thousandpossible pills, while other patients are administered numerous pills ofdifferent types at a given administration time. Accordingly, a systemwhich can identify a type of pill based on a scan of a single unit ofthat type of pill, as well as identify a number of different pills whichare mixed together, can be advantageous.

Embodiments of the devices and methods disclosed herein may be useful atdifferent stages of the process of prescribing and administeringmedications. For example, with reference to FIG. 1, an identificationand/or verification device may be used when medication is dispensed,when dispensed medication is reviewed, and/or when medication isverified at bedside.

For purposes herein, the term “pill” is intended to include any type ofmedication having a solid or semi-solid outer surface which maintainsits shape during normal handling. For example, the term “pill” isintended to include tablets, capsules, caplets, lozenges, suppositories,chewing gum pieces, as well as other types of medication intended forpatient ingestion.

One embodiment of an apparatus 200 configured to collectthree-dimensional data of surfaces of one or more pills is shown inFIGS. 2 a and 2 b. In the front view of FIG. 2 a, a housing 202substantially encloses apparatus 200, and includes a drawer opening 204for the insertion and removal of a pill tray 208. As can be seen in thecross-sectional side view of FIG. 2 b, a drawer 206 includes a pillsupport such as pill tray 208 to support one or more pills 210 withinapparatus 200.

A camera 212 is situated relative to pill tray 208 so that camera 212 isable to capture images from the entire surface of pill tray 208 wherepills may be present. A vibrator (not shown) may be included to vibratepill tray 208 to separate overlapping pills, though in some embodiments,a vibrator may not be present. The vibrator may be in communication witha controller such that the vibrator is only used if requested by thecontroller.

A laser source 216 is positioned such that laser light is directed atpills on pill tray 208 at an angle relative to the camera. In thismanner, triangulation may be used to generate three-dimensional surfacedata. A laser controller 218 controls the direction of a laser beam 220to direct laser light across the pill tray area. In some embodiments, asingle dot of laser light is projected onto the pill tray and sweptsequentially across the pill tray surface. Light reflected from thesurface of pill tray 208 and any pill(s) on pill tray 208 is captured bycamera 212.

A line of laser light may be projected on pill tray 208 and swept acrossthe relevant area in some embodiments. In still other embodiments,two-dimensional patterns of structured light such as a grid pattern oflasers or shadows may be projected on the pill tray surface. In someembodiments, laser source 216 is a Cameo 650 nm, 3 mW laser made byGlobal Laser Ltd., but any suitable laser source may be used. The camerais a DFM22BUC02-ANG camera made by The Imaging Source in someembodiments, though of course any suitable camera may be used. A lens222 for the camera may be a DF6HA-1B made by Fujinon, though anysuitable lens may be used.

Apparatus 200 also may include one or more lights 224 configured toilluminate pill tray 208. This illumination creates shadows of the pillson the pill tray, and images of these shadows may be recorded by camera212 and/or by a separate camera (not shown). Lights 224 may bepositioned so that the pills are illuminated from different directions(either simultaneously or separately), which permits calculation of theshape and size of the pills and/or surface features. Lights 224 mayinclude lights of different colors, including white light, ultravioletlight, red light, green light, blue light, and/or infrared light. Insome embodiments, Raman spectroscopy and/or infrared spectroscopy may beused as part of identifying the pills present on the pill tray.

A scale (not shown) may be included to provide information regardingweight of one or more pills, though in some embodiments, a scale is notincluded.

In alternative embodiments, apparatus 200 may not include a pill tray orany pill support. For example, apparatus 200 may have an open bottomwhich can placed over a group of pills that are resting on a surfacesuch as a table. In some embodiments, the apparatus may be configured asan open device where ambient light is not prevented from reaching agroup of pills to be examined. In some cases, the open device mayinclude a pill support, while in other cases, the open device may notinclude a pill support, and the device may be configured to examinepills that are resting on a table, a tray, or other suitable surface.

FIG. 3 shows a schematic block diagram of a pill identificationapparatus in communication with a controller 302, which in turn is incommunication with a processor 304. Processor 304 may be part of acomputer which includes a display 306 and user inputs 308. The computermay have a memory 310, and may further be connected to a networkcontroller 312. In some embodiments, some or all of the various computercomponents are physically integrated with the pill identificationapparatus such that the apparatus can operate as a standalone unit. Inother embodiments, the pill identification apparatus is connectable toexisting computers, such as a computer at a patient bedside, andtherefore the pill identification apparatus does not necessarily includeeach of the computer components shown in FIG. 3.

An RFID sensor 230 may be included within the pill identificationapparatus as part of a system of identifying pills or identifyingpatients.

Pill Identification and Verification

Turning now to an overall method of identifying pills and verifying thatthe identified pills can be administered to a patient, FIGS. 4 a and 4 bshow a flowchart of one embodiment of such a method. Of course othermethods may be used, including methods which do not include every stepshown if FIGS. 4 a and 4 b, and methods which include different oradditional steps as compared to FIGS. 4 a and 4 b.

A patient is identified in an act 402 so that the correct medicationprescription information can be retrieved from a suitable source, suchas a prescription database within an institution's (e.g., hospital's)information system. This patient identification may be performed usingnames, patient identification numbers, barcodes, or any other suitableprocedure. If the system is in training mode, the act of identifying apatient is not performed. The drug administration information for theidentified patient is acquired from the hospital information system (act404). The nurse locates the medication which has been dispensed andplaces the medication in the medication identification apparatus (act406). Once the medication drawer is determined to be closed, themedication identification apparatus confirms that medication is presentin the apparatus (inquiry 408). Data regarding the pills is thencollected in an act 410, including three-dimensional surface data. Insome embodiments a 3D point cloud is generated (act 412). Based on thecollected three-dimensional surface data and/or the 3D point cloud data,geometric features of the pill(s) are extracted in an act 414. Detailsregarding certain implementations of geometric feature extraction isprovided further below with reference to FIG. 6.

A determination is made whether the system is in training modeidentification mode (inquiry 416). If the system is in training mode,all medications on the pill tray are assumed to be of the same type, andgenerated geometric features are processed to train and calibrate aclassification module (act 418).

If the system is in identification mode, the extracted medicationfeatures are processed by an identification module in an act 420. If theidentification module is able to suitably identify each medication(inquiry 422), the success of identification and/or the medicationidentification information is indicated to the user. In someembodiments, the identified medication can be compared to a list ofprescribed medications (act 424), and the user may receive an indicationas to whether the pill or pills that were placed in the apparatus aresuitable for administering to the patient (act 426). The indication ofsuccessful identification and/or the indication of the suitability ofadministering certain medications may be provided on a display screenthat is part of the identification apparatus. In some embodiments, theoption of a printout may be provided. In still further embodiments,identification information may be sent to a device that is separate fromthe identification apparatus.

Once the identification process is complete, the drawer with the pilltray may be opened automatically by the apparatus (act 428). Theapparatus may include an override input button which allows the user toopen the drawer even if there has not been a successful identificationand/or match with prescription information. Of course, in someembodiments the drawer is manually operated.

In situations where all of the pills have not been identified, or thepills do not match the prescription, error handling may be performed inan act 430. Error handling is discussed below with reference to FIGS. 5a-5 e. If further scanning is required or requested as a result of errorhandling, the method can be returned to inquiry 408 (see FIG. 4 a).

User Interface

Each of FIGS. 5 a-5 f shows a different display/input mode of oneembodiment of a user interface 500 for a pill identification andverification device. FIG. 5 a shows a mode where the device hassuccessfully identified each pill examined by the device, but it hasbeen determined that one or more of the pills should not be administeredto the patient. For example, a processor housed within the device mayhave determined that one of the pills examined by the device is a pillthat was not prescribed, or an extra pill for a prescribed medication ispresent. An image 502 of the pills is displayed to the user, and theoffending pill (or pills) is noted in any suitable manner. The user canselect a “will remove” button 504 and remove the pill. Or, if the useris aware of a recent or revised prescription for the offending pill, theuser can select an “approve” button 506 to override the device andindicate that the pill will be administered to the patient. The useralso may request a reexamination of the pills by selecting a “scanagain” button 508.

FIG. 5 b shows a mode of user interface 500 when the device hasidentified all the examined pills, but has determined that only aportion of one of the pills should be administered according to aprescription. The relevant pill can be noted in any suitable manner onan image 510. The user can acknowledge this information by selecting an“OK” button 512, or, as shown in FIG. 5 a, an “approve” button and/or a“scan again” button may be provided.

FIG. 5 c shows a mode of user interface 500 when the device has failedto locate a pill that should be present according to prescription data.The user is presented with a stock image 514 of the missing pill. Theuser can select a “skip” button 516 to acknowledge and proceed, or theuser can request a rescan by selecting “scan again” button 508.

FIG. 5 d shows a mode of user interface 500 when the device fails toidentify one of the examined pills. An image of the unidentified pill520 is displayed to the user. The user can choose to acknowledge andproceed by selecting an “Ignore” button 522. Or a “Report” button 524can be selected to report the new medication to administrativepersonnel.

FIG. 5 e shows a mode of user interface 500 when the device is unable toconclusively identify an examined medication. Possible matches 526 aredisplayed to the user and the user can select one of the possiblematches. The user can also choose to ignore this event or report theevent.

FIG. 5 f shows a mode of user interface 500 when all of the examinedmedications have been identified and verified as approved for patientadministration. A display 528 of the each of the medications may beshown along with a list 530 of identified and verified. The user mayacknowledge and proceed by selecting “OK” button 512.

Collection of Three-Dimensional Data and Extraction of GeometricFeatures

FIG. 6 is a flowchart 600 of a set of instructions which may be executedby the identification system to collect three-dimensional surface dataand extract geometric features therefrom. Before surface data iscollected, a suitably low level of external light is verified (602). Animage collection act, such as the operation of a laser scanner, isperformed in an act 604. In an act 606, a 3D point cloud is generatedfrom the data collected in act 602. Location of the various pills isperformed in an act 608, and a check for overlapping pills is made in aninquiry 610. If pills are found to be overlapping, the pill tray isvibrated in an act 612 so that the pills separate.

Once all of the pills are found to be not overlapping, data from thegenerated 3D point cloud may be used to calculate the volume of eachpill (act 614). The 3D point cloud contains information about thedistance from the surface of the pill to the surface of the pill tray(or other pill support). Using this information it is possible tocalculate the volume of the pill using standard geometric calculations.The overall shape of each pill also may be determined using the datafrom the 3D point cloud in an act 616. In some cases, the shape of atwo-dimensional projection of the pill may be sufficient for determiningthe identity of a pill, while is other cases, a determinedthree-dimensional shape may be used to identify a pill.

The 3D point cloud also permits the calculation of the distance from thecamera to the surface of the medication, and this information may beused to calibrate the camera and lens assembly to improve focus whencapturing images. A suitable focus for each pill is established in anact 618. Because some pills can be distinguished by analyzinginformation on the surface of the medication, such as letters andnumbers that have been printed on the surface, establishing a suitablefocus for each medication can be important. If pills have differentheights such that the distances between the camera and the pills varysignificantly, multiple images may be captured, with a suitable focussetting for each pill. For example, one image may be recorded for eachheight level which has been identified, and the lens focus may beadjusted for each image.

Color may be used in some embodiments as part of identifying pills. Thecolors captured and analyzed by embodiments disclosed herein may besensitive to the light which illuminates the pills. Exposing the pillsto different lighting during image recordation can provide a more robustcolor analysis. In some embodiments, pills are sequentially exposed todifferent lighting, e.g., first red light, then blue light, and finallygreen light, and a multispectral image or multispectral images arecaptured (act 620). Other sequences and other suitable types of light,including light which is not visible to the human eye (e.g., infraredand ultraviolet), may be used in various embodiments. In someembodiments, a light generator may be used to expose pills to allwavelengths simultaneously, and a high quality camera and can be used torecord the output.

Pill surfaces may include geometric features such as raised areas, orgrooves or other recesses which may be used to identify pills. In someembodiments, the precision of a dense point cloud permits an analysis ofsurface features which can distinguish types of pills from one another.For example, in some embodiments, the width of a dividing line (e.g.score line) may be used to identify a pill.

In some embodiments, directed light may be used to form pill shadows.For example, in an act 622, pills are exposed to light which isprojected from an angle relative to the camera, and thus a shadow isvisible to the camera. Triangulation calculations may be used todetermine the shape and/or size of surface irregularities of the pillsbased on the shadow. In some embodiments, light is sequentiallyprojected from two or more different directions by multiple lightsource, and two or more images are captured.

In an act 624, various further geometric features are extracted from thegenerated information. For example, curvature of pills surfaces and/orfeatures of grooves may be determined from the 3D point cloud. Raised orrecessed inscriptions on pills surfaces may be extracted. An overallarea and/or overall volume of each pill may be generated from the 3Dpoint could. In some embodiments, certain measurements such as diameter,length, width, height, edge curvatures may be determined based on the 3Dpoint cloud.

For each pill, all of the features generated during the method,including geometric features, may be stored for use in identifying thepill. As discussed below, not all of the generated features arenecessarily used to identify each pill, and different features may beused for different pills, even if the pills are examined at the sametime.

Pill Identification

Various methods may be used to identify one or more pills. In someembodiments, a flowchart of rules may be applied to extracted pillfeatures to identify pills. In other embodiments, pill features may becompared to a database of known features for a set of pill types whichcould be matched to the examined pills. In still further embodiments, anartificial neural network may be used to formulate an identificationalgorithm.

Regardless of the method used to identify an examined pill using theextracted pill features, a tiered analysis may be used in someembodiments. With a tiered analysis, a limited set of extracted featuresare analyzed, and if the features can be matched to exactly one pilltype candidate, then it is concluded that a satisfactory match has beenmade and the analysis of the features of that particular examined pillis stopped. If, however, the analysis of a first set of features resultsin more than one possible match, further analysis is conducted. Forexample, a first limited set of extracted features for a scanned pillmay include a color, a pill volume, and a total area of atwo-dimensional projection of the pill. If an analysis of these featuresleads to four candidate matches, further analysis would be conducted.This further analysis may include a review of the presence of a dividingline and the shape of a two-dimensional projection of the pill. Afterthe analysis of this additional second set of features, the fourcandidate matches may be reduced to one, and therefore a match is madeand the pill is considered to be identified. In this manner, computingresources can be efficiently used when identifying the pills.

In some embodiments, pill features, including geometric features, maynot be extracted from a 3D point cloud until such a feature is requestedby a pill identification algorithm or module. For example, a pillidentification algorithm may analyze a first set of geometric featuresand recognize that further features are needed to distinguish from amonga set of candidate pill matches. The pill identification algorithm mayrequest certain features, and the algorithm for extracting thesefeatures may executed on the stored 3D point cloud or other pillexamination data. The newly extracted features are then sent to the pillidentification algorithm.

Pill Verification

Once each of the pills has been identified, the group of pills can becompared to a list of prescribed pills to check whether the identifiedpills are suitable for patient administration.

In some embodiments, identification and/or verification steps may beperformed by a processor which is remote from the device which examinesthe medication to collect three-dimensional data of the pill surfaces.For example, in some embodiments, a laser scanning device may be used tocollect three-dimensional data for a number of pills, and the data maybe sent to a remote processor for analysis. The data may be sentwirelessly, or via a wired network, and/or may include use of theInternet as part of the transmission. The pill identification and/or theverification of the suitability of administering the identified pillsmay be performed by the remote processor, and the results may be sentback to the laser scanning device for display to the user. Or, in someembodiments, results may be sent to a device that is separate from thelaser scanning device.

Exemplary Embodiment

FIG. 7 a is a schematic side view of the interior of one embodiment of amedication identification device 700, showing how a laser generator 702and a camera 704 operate together to capture three-dimensional dataregarding a pill 706 (or multiple pills). Laser light 708 is directed ata pill support surface 710 at an angle relative to the camera positionin order to enable triangulation calculations. In other embodiments, ashadow grid, multi-spectral overlay, or stereoscopic imaging may be usedto collect three-dimensional data regarding pill 706 FIGS. 7 b and 7 cfurther explain this process. The same results can be achieved bysomeone skilled in the art of computer vision by using other knownmethods, such as shadow grid, multi spectral overlay or stereoscopicimaging.

FIG. 7 b is a top view of a single image of pill support surface 710depicting how laser light 708 bends as a result of the shape of pill 706when viewed from the camera viewpoint Using triangulation calculationsand observing only the shape of the laser light as it falls on thesurface of the medication and the medication container, a 3D point cloudof the contents of the medication drawer may be generated using standardgeometric calculations. As the laser light progresses across the pillsupport surface, images of the reflected light are captured by camera704. In some embodiments, the camera captures 60 images per second, andthe laser light passes across pill support surface 710 at a rate of 3 cmper second. Of course, other suitable capture rates and laser lightprogression rates may be used.

FIG. 7 c depicts a composite of multiple images which have been capturedwith the laser moving across the medication surface, the laser lineproviding a set of information to be used for building a 3D profile ofthe medication. The data may be collected by a computer and recorded asdata points within a 3D point cloud. FIG. 7 c shows forty lines,representing the location of the laser as the camera captured fortyseparate images. The location of the medication can be determined byanalyzing the irregularities of the recorded laser lines. Knowing theangle of the laser permits triangulation software to accuratelydetermine surface locations on the pill. As discussed above, features ofthe pill, such as height, shape, etc. can be generated from this surfacelocation data.

FIG. 8 shows a flowchart of instructions which are executed by aprocessor as part of a method 800 of controlling a medicationidentification device to capture three-dimensional data from one or morepills according to one embodiment. The laser is activated (act 802) anddirected at one end of the pill support surface (or other measurementvolume). Image capture is started (act 804) and laser light is movedacross the pill support surface (act 806). In some embodiments, imagerecordation occurs at a regular, predetermined rate. In someembodiments, image recordation may occur at an irregular rate. Forexample, a feedback loop may be used to alter the image capture ratedepending on what type of data is being collected. As the laser lightpasses over areas devoid of pills, a standard image capture rate may bemaintained. Once irregularities such as pills are detected, the imagecapture rate may be increased and/or the laser light movement rate maybe decreased. Once the laser light has covered the entire pill supportsurface, image capture is stopped (act 808) and the laser is deactivated(act 810).

FIG. 9 shows a flowchart describing of instructions which are executedby a processor as part of a method 900 of controlling a medicationidentification device to capture multispectral images. To capturemultispectral images, each of a plurality of lights may be used toilluminate the pill support surface sequentially. When a first light ison, a first image is captured. The first light is turned off, a secondlight is turned on, and a second image is captured. This process cancontinue with any suitable number of lights. FIG. 8 shows one embodimentwhere two light sources are used—a white light source and an ultravioletlight source. In some embodiments, four different light sources are usedwithin the same device: a red light source, a green light source, a bluelight source and an ultraviolet light source.

In method 900, a first, a white light source is activated in an act 902to illuminate the pill support surface and any pills thereon. In an act904, an image is captured with the pill(s) illuminated by the whitelight. The white light source is then deactivated (act 906), and asecond, ultraviolet light source is activated in an act 908. A secondimage is captured with the pill(s) illuminated by the ultraviolet light.The ultraviolet light source is then deactivated in an act 912. Asmentioned above, additional or different light sources may be used aspart of method 900.

FIG. 10 shows a flowchart of a method 1000 of identifying a medicationbased at least in part on features extracted from three-dimensionaldata. In an act 1002, medication features are input into a processor, acomputer storage medium, a network, or any other suitable location wherethe features can be accessed. The medication features may includegeometric features which were generated by analyzing three-dimensionaldata such as a 3D point cloud. Additional features generated duringlaser scanning or other processes may be input. For example, a colorvalue based on data generated by method 900 may be input. In someembodiments, certain features, such as color or general shape, may bemanually input by a user.

In an act 1004, an identification algorithm is executed using at leastsome of the medication features input in act 1002. As mentioned above,in some embodiments, a subset of inputted medication features may beanalyzed, and if a match is not determined, further medication featuresmay be analyzed. The identification algorithm may operate on a singleprocessor or multiple processors. The identification algorithm maycomprise a standardized set of rules which are capable of identifying apill from among a predetermined set of known pills. In alternativeembodiments, the identification algorithm may be configured to compareextracted features to known features of known pills and calculate scoresof which pill or pills most closely match the examined pill. In otherembodiments, the identification algorithm may include a learningalgorithm, as discussed directly below.

The identification of the examined medication is output in an act 1006.

A method 1100 of training a pill identification algorithm is shown inFIG. 11. Medication features are input in an act 1102, and theidentification algorithm in its existing form is executed in an act1104. In an act 1106, the correct pill identification is input to thealgorithm. Based on the result determined by the algorithm in act 1104and the correct pill identification input in act 1106, theidentification algorithm is recalibrated in an act 1108. In embodimentswhere an artificial neural network or other learning program is beingused, the recalibration may be performed automatically by the programitself. In other embodiments, the algorithm may be updated by manuallyrevising the algorithm.

Shift in Reflected Spectrum

In some embodiments, a red laser is used to perform surface scanning ofpills. Due to the organic nature of the pills, the reflected laser lightmay shift slightly within the color spectrum. This shift can result in aportion of the reflected light residing in the blue spectrum, the greenspectrum, or the blue/green spectrum. The light within the blue and/orgreen spectrums forms a narrow line. While the light reflected in thered spectrum may be used for collecting three-dimensional data, in someembodiments the light from the blue and/or green spectrums may be used,in some cases exclusively. Because the strength of reflected light isnot completely uniform across the thickness of a laser line, a narrowline, such as the line reflected in the blue and/or green spectrums, canprovide a more detailed scan of the pills. A method of using light fromonly the blue and/or green spectrum of the reflected light to gatherthree-dimensional surface data may be used to collect data from pills,but also may be used to collect three-dimensional data from otherobjects.

The above-described embodiments can be implemented in any of numerousways. For example, the embodiments may be implemented using hardware,software or a combination thereof. When implemented in software, thesoftware code can be executed on any suitable processor or collection ofprocessors, whether provided in a single computer or distributed amongmultiple computers.

Further, it should be appreciated that a computer may be embodied in anyof a number of forms, such as a rack-mounted computer, a desktopcomputer, a laptop computer, or a tablet computer. Additionally, acomputer may be embedded in a device with suitable processingcapabilities, including a Personal Digital Assistant (PDA), a smartphone or any other suitable portable or fixed electronic device.

Also, a computer may have one or more input and output devices. Thesedevices can be used, among other things, to present a user interface.Examples of output devices that can be used to provide a user interfaceinclude printers or display screens for visual presentation of outputand speakers or other sound generating devices for audible presentationof output. Examples of input devices that can be used for a userinterface include keyboards, and pointing devices, such as mice, touchpads, and digitizing tablets. As another example, a computer may receiveinput information through speech recognition or in other audible format.

Such computers may be interconnected by one or more networks in anysuitable form, including as a local area network or a wide area network,such as an enterprise network or the Internet. Such networks may bebased on any suitable technology and may operate according to anysuitable protocol and may include wireless networks, wired networks orfiber optic networks.

Also, the various methods or processes outlined herein may be coded assoftware that is executable on one or more processors that employ anyone or more of a variety of operating systems or platforms.Additionally, such software may be written using any of a number ofsuitable programming languages and/or programming or scripting tools,and also may be compiled as executable machine language code orintermediate code that is executed on a framework or virtual machine.

In this respect, embodiments of the invention may be embodied as acomputer-readable storage medium or multiple computer-readable mediaencoded with one or more programs that, when executed on one or morecomputers or other processors, perform methods that implement thevarious embodiments of the invention discussed above. Computer readablemedia may include, for example, a computer memory, one or more floppydiscs, compact discs (CD), optical discs, digital video disks (DVD),magnetic tapes, flash memories, circuit configurations in FieldProgrammable Gate Arrays or other semiconductor devices, or othertangible computer storage medium. As is apparent from the foregoingexamples, a computer readable storage medium may retain information fora sufficient time to provide computer-executable instructions in anon-transitory form. Such a computer readable storage medium or mediacan be transportable, such that the program or programs stored thereoncan be loaded onto one or more different computers or other processorsto implement various aspects of the present invention as discussedabove. As used herein, the term “computer-readable storage medium”encompasses only a computer-readable medium that can be considered to bea manufacture (i.e., article of manufacture) or a machine. Alternativelyor additionally, the embodiments of the invention may be embodied as acomputer-readable medium other than a computer-readable storage medium,such as a propagating signal.

The terms “program” or “software” are used herein in a generic sense torefer to any type of computer code or set of computer-executableinstructions that can be employed to program a computer or otherprocessor to implement various aspects of the present invention asdiscussed above. Additionally, it should be appreciated that accordingto one aspect of this embodiment, one or more computer programs thatwhen executed perform methods of embodiments of the present inventionneed not reside on a single computer or processor, but may bedistributed in a modular fashion amongst a number of different computersor processors to implement various aspects of embodiments of the presentinvention.

Computer-executable instructions may be in many forms, such as programmodules, executed by one or more computers or other devices. Generally,program modules include routines, programs, objects, components, datastructures, etc. that perform particular tasks or implement particularabstract data types. Typically the functionality of the program modulesmay be combined or distributed as desired in various embodiments.

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” when used in the claims,shall have its ordinary meaning as used in the field of patent law.

It should also be understood that, unless clearly indicated to thecontrary, in any methods claimed herein that include more than one stepor act, the order of the steps or acts of the method is not necessarilylimited to the order in which the steps or acts of the method arerecited.

Having thus described several aspects of at least one embodiment of thisinvention, it is to be appreciated various alterations, modifications,and improvements will readily occur to those skilled in the art. Suchalterations, modifications, and improvements are intended to be part ofthis disclosure, and are intended to be within the spirit and scope ofthe invention. Accordingly, the foregoing description and drawings areby way of example only.

What is claimed is:
 1. A system comprising: an imaging device configuredto collect surface image data of one or more pills; a controllerconfigured to control the imaging device to collect the image data ofthe one or more pills; the system being configured to generate athree-dimensional point cloud using the surface image data of the one ormore pills; the system being configured to generate geometric data foreach pill from the three-dimensional point cloud; and the system beingconfigured to determine the identity of each of the one or more pillsbased on at least the geometric data generated from thethree-dimensional point cloud.
 2. A system as in claim 1, wherein thesystem is configured to determine the identity of each of the one ormore pills by applying a set of rules to the geometric data for eachpill.
 3. A system as in claim 1, wherein the system is configured todetermine the identity of each of the one or more pills by comparing thegeometric data of each of the one or more pills to a database containinggeometric data associated with a plurality of known pills.
 4. A systemas in claim 1, wherein the system comprises a structured light scannerand the imaging device comprises a first camera configured to captureimages of light reflecting from the one or more pills.
 5. A system as inclaim 4, wherein the structured light scanner comprises a laser lightsource and the first camera is configured to capture images of laserlight reflecting from the one or more pills.
 6. A system as in claim 5,wherein the laser light source comprises a 650 nm laser diode module. 7.A system as in claim 4, wherein the first camera is configured to obtaina visual image of the one or more pills when the pills are fullyilluminated.
 8. A system as in claim 4, wherein the structured lightscanner is configured to generate the three-dimensional point cloud dataregarding surfaces of the one or more pills.
 9. A system as in claim 1,wherein the imaging device comprises a stereoscopic imager.
 10. A systemas in claim 1, wherein the system is configured to determine theidentity of each of the one or more pills additionally based on ameasured pill color.
 11. A system as in claim 1, wherein the system isconfigured to determine the identity of each of the one or more pillsadditionally based on a measured pill weight.
 12. A system as in claim1, wherein the system is configured to determine the identity of each ofthe one or more pills additionally based on determined pill markings.13. A system as in claim 1, wherein the geometric data generated fromthe three-dimensional point cloud comprises pill markings, and thesystem is configured to determine the identity of each of the one ormore pills based on at least the pill markings.
 14. A system as in claim1, wherein the geometric data generated from the three-dimensional pointcloud comprises pill surface texture, and the system is configured todetermine the identity of each of the one or more pills based on atleast the pill surface texture.
 15. A system as in claim 1, wherein thesystem is configured to determine the identity of each of the one ormore pills additionally based on information received from a radiofrequency identification tag.
 16. A system as in claim 1, wherein theidentity of each of the one or more pills comprises an indication of adosage amount contained by each of the one or more pills.
 17. A systemas in claim 1, wherein the system is configured to record the identityand quantity of each of the one or more pills.
 18. A system as in claim1, wherein the system is configured to provide an indication of whetherthe one or more pills should be administered to a patient.
 19. A systemas in claim 1, further comprising a chamber which is configured to beclosed to ambient light.
 20. A system as in claim 1, further comprisinga pill support and a chamber configured to house the pill support.
 21. Asystem as in claim 20, wherein the pill support comprises a tray whichis removable from the chamber.
 22. A system as in claim 1, furthercomprising a second camera configured to obtain a visual image of theone or more pills when the pills are fully illuminated.
 23. A system asin claim 1, wherein the one or more pills comprise at least one of apill, a capsule, tablet or a caplet.
 24. A system as in claim 1, whereinthe one or more pills comprise a plurality of pills, and a first pill ofthe plurality of pills contains a first medication which is differentfrom a second medication contained in a second pill of the plurality ofpills.
 25. A system as in claim 1, wherein the one or more pillscomprise a single pill.
 26. A method comprising: collecting surfaceimage data of one or more pills; generating a three-dimensional pointcloud of surface data for each of the one or more pills; generatinggeometric data from the three-dimensional point cloud data; using acomputer processor to identify the one or more pills based on at leastthe geometric data generated from the point cloud data.
 27. A method asin claim 26, wherein collecting surface image data comprises operating astructured light scanner to collect surface image data of the one ormore pills.
 28. A method as in claim 27, wherein operating a structuredlight scanner comprises applying laser light to surfaces of the one ormore pills and using a first camera to capture images of laser lightreflecting from the surfaces of the one or more pills.
 29. A method asin claim 28, wherein applying laser light comprises using a 650 nm laserdiode module.
 30. A method as in claim 26, collecting surface image datacomprises operating a stereoscopic imager to collect surface image dataof the one or more pills.
 31. A method as in claim 26, whereinidentifying the one or more pills comprises applying a set of rules tothe geometric data for each pill.
 32. A method as in claim 26, whereinidentifying the one or more pills comprises comparing the geometric dataof each pill to a database containing geometric data associated with aplurality of known pills.
 33. A method as in claim 26, furthercomprising comparing the identities of each of the one or more pills toa list of pills approved for administration to a patient.
 34. A methodas in claim 26, further comprising recording the identity and quantityof each of the one or more pills.
 35. A method as in claim 26, whereinthe geometric data includes geometric data regarding a dividing line ona pill, and identifying the pill comprises identifying the pill based onat least the geometric data regarding the dividing line.
 36. A method asin claim 35, wherein the geometric data includes a thickness of thedividing line on a pill, and identifying the pill comprises identifyingthe pill based on at least the thickness of the dividing line.
 37. Amethod as in claim 26, wherein the geometric data includes geometricdata regarding surface texture of the more or more pills, andidentifying the one or more pills comprises identifying the one or morepills based on at least the surface texture.
 38. A method as in claim37, wherein identifying the one or more pills comprises identifying theone or more pills additionally based on a measured pill color.
 39. Amethod as in claim 26, wherein identifying the one or more pillscomprises identifying the one or more pills additionally based on ameasured pill weight.
 40. A method as in claim 26, wherein identifyingthe one or more pills comprises determining a dosage amount of each ofthe one or more pills.
 41. A method as in claim 26, wherein the one ormore pills comprise a plurality of pills, and a first pill of theplurality of pills contains a first medication which is different from asecond medication contained in a second pill of the plurality of pills.42. A method of performing surface scanning, the method comprising:projecting a laser light pattern onto a three-dimensional surface, thelaser light comprising laser light within at least one of the red andinfrared spectrum; receiving, with a camera, laser light reflected froma portion of the three-dimensional surface; and using light only from atleast one of a green and blue spectrum of the reflected light todetermine the location of the portion of the three-dimensional surfacefrom which the reflected light reflected.
 43. A method as in claim 42,wherein projecting a laser light pattern onto a three-dimensionalsurface comprises projecting the laser light pattern onto a pill surfaceto generate three-dimensional scanning data, and the method furthercomprises: generating geometric data from the three-dimensional scanningdata; and identifying the pill based on at least the geometric datagenerated from the scanning data.